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ABSTRACT 

This document examines the following aspects of the 
federal administration of evaluation, program improvement, and 
technical assistance under Chapter 1 of the Education Consolidation 
and Improvement Act: (1) changes in each of the three areas since 
1981 when Chapter 1 replaced Title I of the Elementary and Secondary 
Education Act; (2) activities currently conducted in the United 
States Department of Education (ED) in each area; and (3) problems 
and opportunities experienced in the federal administration of 
evaluation, program improvement, and technical assistance. Among the 
conclusions are the following: (1) ED has encouraged state and local 
educational agencies (SEAs and LEAs) to continue collecting data on 
project effectiveness and participant characteristics using the 
evaluation system mandated under Title I; (2) reports of errors in 
the research design, collection, and reporting of achievement data 
suggest that ED should assess the quality of those data to determine 
if changes are needed; (3) problems in the federal administration of 
the improvement initiative arise from the voluntary nature of state 
and local activities to improve Chapter 1 quality, that is not the 
SEAs' central purpose; and (4) Technical Assistance Center 
performance in assisting SEAs and LEAs in program improvement has 
been limited by constraints on their resources, low levels of 
interest, and the structure of the Chapter 1 program. Data are 
presented on three tables; a list of references is included. (BJV) 
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EXECUTIVE SUMMARY 

This study examined the federc'l acjninistration of evaluation, 
program improvement, and technical assistance undsr Chapter 1 of the 
Education Consolidation and Improvement Act (ECIA), In particular, 
it reviewed (1) changes in each of the three areas since 1981 when 
Chapter 1 replaced Title I of the Elementary and Secondary Education 
Act, (2) activities the U,S, Department of Education (ED) currently 
conducts in each area, and (3) problems and opportunities experienced 
in the federal administration of evaluation, program improvement, and 
technical assistance. 

These federal activities were reviewed in a single study because 
they share certain important features. The most important is their 
capacity to shape and change Chapter 1 services at the local level, 
even though the relevant statutory provisions impose rt^latively few 
actual requirements on state and local educational agencies (SEAs and 
LEAs). Unlike other areas of Chapter 1 administration, federal activi- 
ties in evaluation, program improvement, and technical assistance are 
intended to provide SEAs and LEAs with encouragement, ideas, and 
practical methods, not just mandates, for assessing and improving the 
effectiveness of Chapter 1 services. 

Federal Administration of Chapter 1 Evaluation 

Chapter ' ^ as amended, requires SEAs and LEAs to evaluate their 
Chapter 1 services, although ECIA allows educational agencie .-greater 
discretion in the design of their evaluations than was permitted 



report information on the characteristics of students receiving Chapter 
1 services, ED has encouraged agencies to continue collecting data 
on project effectiveness and participant characteristics using the 
evaluation system mandated under Title I. Federal program review 
teams monitor state implementation of Chapter 1 evaluation procedures 
(although this information is not routinely fed back to the ED office 
with lead responsibility for program evaluation). For the most recent 
reporting cycle, A5 of the 53 SEAs submitted Chapter 1 achievement 
data that could be aggregated and analyzed nationally, and all SEAs 
submitted at least some data on Chapter 1 participants. 

Reports of errors in the research design, collection, and reporting 
of Chapter 1 achievement data suggest that ED should assess the quality 
of those data to determine if changes are needed. In light of the 
federal uses of Chapter 1 achievement and participation data. Congress 
and ED should also assess how these data are being used by SEAs and 
LEAS, to dete'Miine if (1) changes are needed to ensure greater accuracy 
and to make the data more relevant to state and local administrative 
needs and (2) the leveD of use justifies the federal, state, and 
local expenditures for operation of the current data collection and 
reporting system. (For example, the low reported use of mandated age 
and gender data on Chapter 1 participants suggests that these data 
requirements should be reviewed,) 
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Federal Spcnsorship of Program Improvement Under Chapter 1 

A major new initiative in Chapter 1 since 1981 is the iinplementa- 
tion of activities intended to promote instructional improvement in 
Chapter 1 projects. This initiative has had three purposes, includ- 
ing (1) the general upgrading of Chapter 1 instruction, based on 
principles described in the "effective schools" research; (2) the 
provision of intensive assistance to selected Chapter 1 projects that 
have particularly poor records of performance; and (3) the use of 
evaluation to identify needs and suggest strategies for improving the 
achievement of Chapter 1 participants. 

ED has implemented this initiative through (1) the award of 
small grants to SEAs for designing and disseminating approaches to 
improve project quality, (2) the identification and recognition of 
246 projects judged to provide particularly successful Chapter 1 
services, and (3) increases in the availability of technical assistance 
for improving the quality of Chapter 1 services. 

Problems in the federal administration of the improvement initia- 
tive arise from the voluntary nature of state and local activities to 
improve Chapter 1 quality. Because upgrading Chapter 1 instruction 
has never been a central purpose of the state administrative mission, 
SEAs rnay be reluctant to commit scarce resources to it, especially 
since this new federal emphasis has not reduced their responsibilities 
for ensuring local compliance with Chapter I's legal requirements. 
Limits on federal and state resources available for improvement have 
meant that much of the impetus for these activities must come from 
the broader educational ref ^m movement, which states and school 



systems have adopted in varying ways and with varying levels of inten- 
sity, A major challenge confronting federal Chapter 1 administrators 
is to find ways to encourage linkages between the Chapter 1 improve- 
ment initiative and state efforts to achieve broader reform objectives. 

Federal Provision of Chapter 1 Technical Assistance 

In addition to providing technical assistance in interpreting 
and complying with program requirements, ED delivers technical assis- 
tance through its sponsorship of the Technical Assistance Centers 
(TACs), which are currently funded at an annual total of $3.6 million. 
Although initially established to assist in the adoption and implemen- 
tation of the Title I evaluation and reporting system, the TACs now 
also have major responsibilities for the implementation of the Chapter 
1 program improvement initiative. Indeed, this study found that the 
TACs serve as the main link between ED and state and local practitioners 
in promoting and implementing both program improvement and evaluation. 

Current evidence indicates that the TACs are implementing the 
program improvement responsibilities assigned to them, as well as 
their earlier assigned responsibilities for evaluation. TAC perfor- 
mance in assisting SFAs and LEAs in program improvement has been 
limited, however, by (1) constraints on their resources and on state 
funds available to carry out program improvement activities, (2) low 
levels of interest and commitment to improving Chapter 1 programs in 
some states, and (3) the categorical structure of the Chapter 1 program, 
in which only certain students are served within a school and not the 



whole school. Despite these limitations, however, Chapter 1 coordina- 
tors indicate satisfaction with the quality of the TACs' assistance in 
both evaluation and program improvement. 
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I. INTRODUCTION 

This study examined the federal administration of evaluation, 
program improvement, and technical assistance under Chapter 1 of the 
Education Consolidation and Improvement Act (ECIA). In particular, 
the study sought to identify and describe the federal administrative 
methods that are used in these three areas to advance the goals set 
for the implementation of the Chapter 1 basic grants program. 

This section of the report describes (1) the purposes of the 
study; (2) the research design; (3) the legal framework for evalua- 
tion, program improvement, and technical assistance; and (^) the 
organization of the report. 

Purposes of the Study 

This is one of four studies of Chapter I's federal administra- 
tion conducted for the National Assessment of ECIA Chapter 1. The 
other three studies review (1) selected aspects of Chapter I's legal 
framework, (2) federal staffing and budgeting policies affecting the 
administration of Chapter 1, and (3) federal compliance and oversight 
activities under Chapter 1. 

The descriptive information reported in this study supplements 
other information developed for the National Assessment of ECIA 
Chapter 1 in two respects. First, in conjunction with the studies 
just described, it describes the most important federal administra- 
tive activities under Chapter 1. Second, it complements information 
on Chapter 1 evaluation, program improvement, and technical assis- 
tance contained in the Assessment's studies of Chapter 1 
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implementation at state and local levels. (See especially Farrar & 
Millsap^ 1986, and Knapp, Turnbull, Blakely, Jay» Marks, & Shields, 
1986J 

Witii a few exceptions, this report does not describe the state 
and local implementation of federal requirements and priorities in 
evaluation, program improvement, or technical assistance. In order 
to obtain a complete picture of the operation of these activities 
under Chapter J» it would be necessary to axamine our data and find- 
ings in combination with the information obtained in the Chapter 1 
studies that focused on state and local operations. 

Federal activities in evaluation, program improvement, and tech- 
nical assistance can shape and change Chapter 1 services, even though 
the statutory provisions authorizing these activities are not as 
prescriptive as other Chapter 1 components. Insteao, these federal 
activities are intended to provide state and local educational agen- 
cies (SEAs and LEAs) with encouragement, ideas, and practical 
methods, not just mandates, for assessing and improving the effec- 
tiveness of Chapter 1 services. 

To learn more about these federal activities, the study asked 
several general questions about each of the three functional aioas; 

• Wliat changes have occurred in each area of 
federal administration since 1981? 

• What activities does the U.S- Department of 
Education (ED) currently carry out in each area? 

• What opportunities and problems do federal 
respondents experience in connection with their 
work in each area? 

This report presents the answers we found to these questions. 




Study Design 

The research design we used in the study had three parts: 

1 . Review of research literature and other data 

We examined articles and reports describing federal requirements 
and practices in evaluation, program improvement, and technical 
assistance since the enactment of Tit' a I of the Elementary and 
Secondary Education Act (ESEA), the predecessor of ECIA Chapter 1. 
In particular, we looked for information on the strategies adopted to 
implement each area and the priority attached to each in the overall 
administration of the federal compensatory education program. Among 
the documents that we reviewed wei'e reports on the design and imple- 
mentation of the Title l/Chapter 1 Evaluation and Reporting System 
1;TIERS) and the Title I/Chapter 1 Technical Assistance Centers 
(TACs). 

2 . Interviews with persons responsible for Chapter 1 federal 
administration 

Among the 36 persons interviewed for the study were 25 federal 
officials (including congressional staff members) involved in the 
administration or oversight of the areas under review. In addition, 
we interviewed the directors of all four TACs, heads of the TAC area 
offices, and several state Chapter 1 coordinators. We also talked 
with other persons who have either conducted research in the areas of 
interest to us or have first-hand familiarity with the topics 
addressed in this study. 

3 . Data analysis 

We analyzed the information obtained through interviews and the 
review of documents in lignt of the three questions listed earlier. 

.?3 
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In several instances we constructed longitudinal analyses to deter- 
mine how certain activities had changed over time. Finally, we 
integrated the results of all analyses to develop the study's conclu- 
sions • 

Current Legal Framework for Federal Administrative Activities in 
Evaluation. Program Improvement, and Technical Assistance -^ 

The Chapter 1 law and regulations establish the structure for 
federal activities in evaluation, program improvement, and technical 
assistance- The requirements in each of these areas are described 
below. 

Evaluation Provisions 

More detailed federal requirements apply to evaluation the i to 
improving program quality or providing technical assistance. ECIA 
Chapter 1 requires SEAs to evaluate programs receiving Chapter i 
assistance. Section 555(e) states that each SEA shall: 

(1) conduct an evaluation of the programs assisted 
under this chapter at least every two years and 
shall make public the results of that evaluation; 
and 

(2) collect data on the race, age, and gender of 
children served by the programs assisted under 
this chapter and on the number of children served 
by grade level under the programs assisted under 
this chapter. 

In addition, as stated in Section 555(d), each SEA must ''keep 
such records and provide such information to the Secretary as may be 



^ This discussion of legal requirements is based, in part, on 
the report on selected aspects of the Chapter 1 legal framework 
(Gaffney & Schember, in draft, January 1987). 
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required for fiscal audit and program evaluation." Section 591(b) of 
ECIA , however , prohibits the Secretary from issuing regulations 
"relating to the details of . . . evaluating programs and projects by 
state and local educational agencies." 

In response to Section 555(e), Chapter 1 regulations essentially 
repeat the statute (as cited in the next paragraph) in their direc- 
tion that states may fulfill the evaluation requirements by 
aggregating evaluation data from LEAs ' "obj ective measurements of 
educational achievement in basic skills" and reporting statewide 
totals. The regulations also reiterate the statutory requirement 
that SEAs must annually collect and report data on Chapter 1 partici- 
pants' race, age, gender, and grade level. Although the statute does 
not specify the frequency of this data collection and reporting, the 
regulations require that the data be obtained and reported annually. 
(As described in Section II of this report, these participant data 
are to be reported to ED in a specified format.) 

Section 556(b)(A) of the statute states that an LEA receiving 

Chapter 1 assistance must assure that its programs and projects: 

. . . will be evaluated in terms of their effectiveness in 
achieving the goals set for them, and that such evaluations 
shall include objective measurements of educational 
achievement in basic skills and a determination of whether 
improved performance is sustained over a period of more 
than one year; and that the results of such evaluation will 
be considered by such agency in the improvement of the 
programs and projects assisted under this chapf^^r. 

The regulations require that an LEA conduct such an evaluation at 

least once every three years. Additional requirements for evaluation 

are imposed on Chapter 1 schoolwide projects. 
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By cross-referencing the LEA statutory requirements in the 
federal regulations on SEA evaluations , ED is suggesting — but not 
mandating — that SEAs meet their statutory evaluation requirement by 
aggregating LEAs' "objective measurements of educational achieve- 
ment*" Since an LEA must conduct such an evaluation only once every 
three years, an SEA that bases its own (valuation on LEA data must 
coordinate LEA schedules statewide, in order to be certain that its 
own biennial report contains the achievement results from a repre- 
sentative sample of Chapter 1 programs. 

The Chapter 1 1983 Noiireg-alatory Guidance (NRG) states that SEAs 
will "approv[e LEAs'] evaluation designs that meet Chapter 1 require- 
ments" and that the "models described in [Title I regulations] are 
appropriate evaluation designs, • • • [but] the Department does not 
require the use ..f those, or any other, particular models. ""^ The NRG 
issued in 1986 also states that SEAs, to meet their own evaluation 
obligations, "may wish to require that • • • LEAs use specific 
instruments to measure achievement or specific evaluation designs," 

Program Improvement Provisions 

Encouragement for improving Chapter 1 programs is a feature of 
Section 556(b) (A), cited above, requiring LEAs to consider evaluation 
results in improving their project activities • In addition. Section 
556(b)(3) of the law requires LEAs to assure that their Chapter 1 
programs "are of sufficient size, scope, and quality to give reason- 
able promise of substantial progress toward meeting the special 



^ The Title I evaluation models are described in Section II of 
this report. 
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educational needs of the children being served. • . J* These state- 
ments constitute the legislative authority for federal and state 
attention to instructional quality issues. 
Technical Assistance Provisions 

Section 591 of ECIA, which limits the Secretary's regulatory 

authority as cited above, also states: 

[I]n all other matters relating to the details of planning, 
developing, implementing, and evaluating programs and proj- 
ects by state and local educational agencies the Secretary 
. - • may • • • 5 upon request , provide technical assis- 
tance, information, and suggested guidelines designed to 
promote the development and implementation of effective 
instructional programs and to otherwise assist in carrving 
out the purposes of this subtitle. 

Organization of the Report 

This report contains four additional sections. Section II 
describes the federal administration of Chapter 1 evaluation activi- 
ties conducted at state and local levels. Section III presents 
information on Chapter I's federally administered activities in pro- 
gram improvement. Section IV describes the federal administration of 
Chapter 1 technical assistance activities • Section V summarizes the 
conclusions of the study. 
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II. FEDERAL ADMINISTRATION OF CHAPTER 1 EVALUATION 



Program and project evaluation have been central features of the 
federal compensatory education program since the 1965 enactment of 
Title I, when Sen. Robert F. Kennedy added the requirement that LEAs 
evaluate the effectiveness of their projects in serving educationally 
deprived children. In the intervening years, federal responsibili- 
ties in evaluation have experienced several shifts, resulting in the 
current federal administrative role that is in many ways a hybrid of 
what has gone before. Federal requirements for state and local 
evaluation have focused on two general types of information — the 
achievement outcomes of Chapter i students and certain descriptive 
data on Chapter 1 participants, including the subject areas in which 
they receive Chapter 1 services, numbers of Chapter 1 participants 
enrolled in public and nonpublic schools, and participants' grade 
level, racial/ethnic group affiliation, age, and gender. 

In this section, we describe (1) the major changes since 1981 in 
the federal administration of Chapter 1 evaluation, (2) the evalua- 
tion activities ED currently carries out, and (3) the opportunities 
and problems characterizing this area of federal administration. 

Changes Since 1981 in the Federal Administration of Chapter 1 Evalua- 
tion 

ECIA*s statutory provisions signaled important changes in the 
federal administrative role in evaluating federally assisted compen- 
satory education services. The 1983 ECIA Technical Amendments and ED 



actions in administering Chapter 1 have, however, combined to lessen 
the effect of the changes mandated in 1981. 
Legislative Mandate for Change 

The philosophy of administrative simplification embodied in ECIA 
affected evaluation as much as any other area of Title I administra- 
tion. Most importantly, ECIA repealed the Title I requirements (1) 
that SEAs and LEAs use the evaluation models (known as the RMC 
models , denoting the contractor responsible for their development ) 
that had been mandatory since the 1979-80 school year^ and (2) that 
SEAs report evaluation results using the TIERS format and metric.'^ 
In addition, ECIA repealed the Title I requirement that the federal 
government provide technical assistance to SEAs **to enable them to 
assist LEAs in the development and application of a systematic lu- 
ation of programs in accordance with the models" (Section 183(e) of 
ESEA Title I, as amended). 

ECIA also eliminated or changed other evaluation related pro- 
visions that the 1983 ECIA Technical Amendments later reimposed. 
These included the requirements that LEAs use evaluation results to 
improve their Chapter 1 activities and that SEAs collect evaluation 

^ School year 1981-82 was the final year that tb^ law required 
SEAs to collect acliievement data using the RMC models and the TIERS format. 

^ Under TIERS, SEAs annually collect and report the following 
information by grade level and subject area (i-e., reading, mathe- 
matics, and language arcs): number of students tested, nuiriber of 
students with both pretest and posttest scores, test interval, and 
post test scores and gains . Gains are reported using a standard 
reporting metric developed for TIERS — the normal curve equivalent 
(NCE). Use of this standard metric permits scores to be aggregated 
across tests, districts, and states. 
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data from Chapter 1 projects within their states. The Technical 
Amendments also restored the requirement that states collect informa- 
tion on the grade level and racial/ethnic characteristics of Chapter 
1 participants, information that had been included in the TIERS 
reports before 1981; in addition, the Amendments also required that 
SEAs collect data on participants' age and gender, which had not been 
required under Title I. 

The Technical Amendments, however, did not reimpose the manda- 
tory use of the RMC models or the TIERS reporting format. 

Federal Implementation of ECIA 

ED implemented the requirements of ECIA, including those of the 
1983 Technical Amendments. As described in the following pages, the 
administrative actions taken by ED (directed by U.e Department's 
Planning and Evaluation Service — PES) indicated assumptions that (1) 
ECIA was not likely to reduce congressional interest in monitoring 
the effectiveness of Chapter 1 and (2) the new law would not diminish 
the utility of state and local assessments regarding the outcomes of 
their compensatory education services. So, with the encouragement of 
groups such as the Chapter 1 state coordinators, PES and ED's Compen- 
satory Education Programs (CEP) staff urged SEAs and LEAs to continue 
using the Title I evaluation system. 

ED'S Current Administrative Activities in Chapter 1 Evaluation 

ED'S main current activities in the administration of state and 
local Chapter 1 evaluation activities are (1) encouraging the con- 
tinued use of the RMC models and the TIERS reporting format, (2) 
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t*ponsoring TAG services, (3) monitoring the state and local implemen- 
tation of the Chapter 1 evaluation requirements (including the use of 
evaluation results to improve instructional services), and (A) 
analyzing and reporting states * Chapter 1 participation and achieve- 
ment data. These activities and the current ED administrative 
structure for evaluation are described below, followed by a discus- 
sion of ED'S use of Chapter 1 evaluation ds^a. 

Responsibilities for federal administrative activities in evalu- 
ation are divided betv;een PES and CEP. Because PES staff supervised 
the design of TIERS and the RMC models and continue to have lead 
responsibility for monitoring the TACs, PES ays a key role in 
Chapter 1 evaluation. CEP*s role in Chapter 1 evaluation has grown 
in recent years, with the adoption of the Chapter 1 program improve- 
ment focus and the development of its in-house capacity to conduct 
quantitative analyses of states* evaluation and participation data. 

Under current procedures, ED asks (but does not require) SEAs to 
submit annual statewide Chapter 1 ac lievement information in essen- 
tially the same categories used in TIERS. ^ ED*s form (No. 686-2) for 
Chapter 1 state performance reports includes the following statement: 

The Department is interested in . . . grade level, number 
cf students tested, number of students with pretest and 
posttest scores, test interval . . ., and posttest scores 
and gams, both reported according to some standard metric 
(e.g., normal curve equivalents). 



^ Some states also send ED copies of their state Chapter 1 
evaluation reports, which must be prepared every two years. When 
asked about their use of these reports, ED staff said the reports are 
examined when specific questions arise about a particular state. 
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The instructions note that "states which are continuing to aggregate 
achievement data according to the former Title I evaluation reporting 
system w,^y submit such data aggregations," 

The instruction.^ do not advise SEAs on how they should coordi- 
nate their annual performance reports with LEAs' requirement to 
evaluate their programs once every three years. The ED reporting 
form does, however, ?sk the SEA if the achievement data are represen- 
tative of the state ad a whole. If the SEA responds that the data 
are representative, it is asked to provide a copy of the sampling 
plan. If not, it is ask^d to "provide a brief explanation of data 
limitations ." 

Under federal direr "ion, the TACs continue to assist states in 
generating these data, as tney have si":ce 1976, despite the repeal of 
the Title I provision requiring federally sponsored technical assis- 
tance in evaluation. Although th TAG mission has expanded to 
include a new focus on Chapter 1 program improvement (as discussed in 
Sec^-ion III of this report), lAC directors interviewed in this study 
indicate that at least hplf of the TACs' resources continue to be 
used for evaluation assista^^ce. 

Respondents from CEP r^iport that they monitor SEAs' implementa- 
tion of Chapter 1 evaluation requirements in their state program 
reviews* CEP's current program review g"aide (dated August 1985) 
addresses evaluation in several contexts, as follows: 

• In the review of an SEA's monitoring of LEA 
applications and repo""ts , program review teams 
are instructed to look for problems such as 
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— no penalties for late or missing 
reports and 

— incomplete records. 

• In the review of an SEA^s evaluation procedures 
and records, they are to look for problems such 
as 

— lack of plans to develop and publish 
the state evaluation report, 

— no feedback to LEAs concerning their 
evaluations , 

— faulty interpretations or inappropriate 
use of data, 

— failure of LEAs to use evaluation 
results for program improvement, and 

— few sustained effects studies by LEAs. 

The November 1985 summary of state program reviews, the most 
recent available, reports on CEP findings, recommendations, and cor- 
rective actions. Program reviewers found a wide range of problems in 
evaluation procedures and findings. At the SEA level these problems 
included the following: 

e Lack of evidence that the SEA is using sustained 
gains information to target program improvement 
activities , 

• Unexplained declines ir third-grade test scores, 

• LEAs' use of outdated forms of standardized 
tests , 

• Improper sampling of LEAs, and 

• Lack of systematic collection and use of evalua- 
tion data as a basis for SEA technical assistance 
on program improvement. 

They noted the following problems at the LEA level: 

• Lack of satisfactory explanation for decreases in 
test scores. 



• Lack of satisfactory explanation for unrealis- 
tically high gains in achievement scores, 

• Use of fall-to-spring testing rather than annual 
cycles, and 

• Excessive expenditures for outside evaluators. 

PES staff said that none of these problems had been brought to their 
attention nor communicated to the TACs, indicating a need for better 
communication between CEP and PES. 

ED'S current activities concerned with the analysis and report- 
"'ng of states* Chapter 1 achievement and participant data reflect an 
interest in using these data to improve state Chapter 1 procedures. 
When CEP receives the state performance reports each year, staff com- 
pare the data contained in each report to tho previous year's 
achievement and participant data submitted by the state, highlighting 
significant changes. These comparisons are sent to each state for 
their information, allowing the state to obtain immediate feedback 
and to correct any errors in its report thuC may have prompted an 
erroneous identification of "significant changes.** Following this 
preliminary analysis, the state reports are forwarded to a pri.vate 
firm for more comprehensive analysis . The contractor prepares a 
national summary of the achievement and participation data and a 
report for each state , displaying the staters trends in Chapter 1 
participation and achievement outcomes over time. (The most recent 
national summary report is Gutman & Henderson, Febriiary 1987 [in 
draft], which presents achievement and participation data for school 
year 198A-85.) CEP encourages SEAs to use the reports as a basis for 
assessing their own performance (by comparing the state's performance 



trends to the nation^s) and also as a basis for feedback to LEAs (by 
comparing each LEA^s performance trends to the state's). 

ED uses these evaluation data in several ways. In addition to 
assisting states to assess their Chapter 1 performance, ED also uses 
analyses of the TIERS data to reveal national patterns of Chapter 1 
implementation and effects. Recently, the staff of the National 
Assessment of ECIA Chapter 1 used TIERS analyses as a basis for 
describing the effects of Chapter 1 services on the reading and 
mathematics achievement of Chapter 1 students (Kennedy, Birman, L 
Demaline, 1956 )• In addition, PES staff regularly include analyses 
of TIERS achievement and participation data in ED^s annual evaluation 
report. (The most recent is cited as Office of Planning, Budget and 
Evaluation, FY 1985.) PES and CEP staff interviewed for this study 
said that TIERS data are also used in (1) preparing congressional 
testimony, (2) drafting speeches for delivery to national and local 
audiences , and ( 3 ) bri.ef ing CEP program review teams before they 
visit specific states. 

Qm^or^u nities and Problems in the Federal Administration of Chapter I 
Evaluation 

Current issues in the administration of Chapter 1 evaluation 
stem mainly from the uncertain purpose underlying these activities . 
Is ED encouraging states to continue collecting TIERS data with an 
intent that the aggregated data will preoent a national picture of 
the effectiveness of Chapter 1? Are the data intended mainly as a 
basis for SEAs * and LEAs* self-assessment and henc5 as a component of 



federal efforts to promote Chapter 1 program improvement? Or are the 
two purposes equally important? 

If the two purposes are equally important, the TIERS design may 
be the best approach for addressing the two goals through a single 
system. At the same time, other options are available to ED for 
meeting each goal. The following discussion outlines several prob- 
lems in the current Chapter 1 evaluation system an'' notes possible 
alternatives to TIERS. 

Pr oblems in the Collection and Completeness of Evaluation Data 
Obtained bv SEAs 

The importance of reassessing federal policies in Chapter 1 
evaluation is borne out by current trends in the SEAs' collection and 
reporting of Chapter 1 participation and achievement data and in the 
quality and completeness of the data submitted to ED. 

P >ol year 1979-80, the first year SEAs were required to report 
\ > , d, was the only year in which all 53 SEAs (50 states plus 
the District of Columbia , Puerto Rico, and the B; repu of Indian 
Affairs) reported participation and achievement data using the TIERS 
format . This high level of compliance reflects the fact ♦'.hat the 
TACs, the TIERS developers, and federal staff worked with SEAs for 
two years to prepare for the first year of mandatory data collection. 

• For the 198^-85 school year, the most recent year for which data 
have been submitted to ED, all 53 SEAs reported state participation 
data '.o ED. .<^lthough m?ny states continue to report their achieve- 
ment u.'- to FD uFing the TIERS format, their number has decreased: 
45 of th 53 SEAs reported achievement data in a format capable of 
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being analyzed and aggregated at the national level. ^ This figure is 
the lowest number of SEAs reporting usable Chapter 1 information 
(except for 1981-82, when ED did not ask SEAs to submit evaluation 
data). Additionally, of the A5 SEAs, many did not report outcome 
data at all grade levels at which Chapter 1 services were provided. 

Evidence of problems in the quality and completeness of the data 
is growing. Based or his interviews with Chapter 1 state coordina- 
tors, Dougherty reports that "the most difficult problem in evalua- 
tion appears to be that of producing valid results and recognizing 
when results are not valid" (1986, p. 151). Indeed, virtually all of 
the federal staff involved in the collection and analysis of Chapter 
1 achievement and participation data report examples of serious prob- 
lems.^ For ex .iple: 

• One state combines reports on all grade levels , 
making it impossible to separate average 
achievement gains for the primary grades from 
those for other grades. 

• Another very small state aggregates data from 
only a sample of its districts, even though the 
state 's small total number of districts techni- 
cally precludes any sampling. 



^ Georgia and Connecticut reported no achievement data. Data 
reported from the following SEAs could not be compared with data from 
other states: Puerto Rico used Spanish-language tests that differed 
from English-language achievement tests used by the other SEAs; the 
District of Columbia reported test results in grade equivalents 
rather than NCEs; Oklahoma provided data at only three grade levels; 
and Maine, New Hampshire, and Maryland did not report po ntest scores. 

^ Personnel in one ED staff office reported that they do not 
use the achievement data from SEAs in their Chapter 1 analyses 
because of doubts about the accuracy of the data. They said that the 
data may be useful for comparisons over time within a state but not 
for comparisons among states or aggregation across states. 



• In a niimber of states, districts shift from one 
evaluation cycle to another at their own discre- 
tion, without coordinating their data collection 
with the state's overall sampling plan. 

• One state did not collect Chapter 1 data on the 
race of Chapter 1 participants, so it used the 
racial proportions that a neighboring state 
reported on its Chapter 1 application. After 
exaimining the results, SEA officials realized 
that they probably had more Asian- American 
students enrolled in Chapter 1 projects than 
their "borrowed" estimates reflected, so the SEA 
used guesswork to increase its reported enroll- 
ment of Asian-Americans, 

• Another state that has one large urban district 
obtains participation data from the district 
every other year and does not compensate for the 
omission of the district in the off years • As a 
result, the reported Chapter 1 participation 
figures vary widely from one year to the next. 

Even more pervasive than these problems are instances of care- 
lessness in the reporting of Chapter 1 data. According to ED staff 
who regularly review the data SEAs submit, some SEA reports look as 
though they are not* checked for accuracy or reasonableness (e.g., 
columns may not equal the reported total or totals may be unreason- 
ably high or low) or for correct transcription (e.g., numbers may be 
transposed) . ^ 

These reviewers note that, although most of these problems have 
come to light relatively recently, they have no way of knowing 
whether the problems themselves are recent or only recently dis- 
covered. CEP staff point out that the problems arising in SEA and 
LEA evaluations tend to be less serious than those arising several 



^ An observation pertinent to this review comes from one of the 
TAC directors who said that the data that LEAs send to SEAs are 
generally of good quality. According to this respondent, the errors 
and sloppiness occur at the SEA level. 
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years ago. In any case, these problems build a strong argument for 
conducting a thorough critique of the overall quality of the Chapter 
1 achievement data. This critique could also usefully examine the 
Chapter 1 participation data, even though fewer problems have come to 
light from those reports. If such a review indicates that the 
quality of the data is acceptable, a second review shoula be con- 
ducted of SEAs' and LEAs' use of the data to assess and improve their 
Chapter 1 programs. Information bearing on this review is available 
from several of the state and local case studies conducted for the 
National Assessment of Chapter 1. The findings of these reviews 
should be used to indicate areas of needed change in the Chapter 1 
evaluation system. 

Issues Concerning the Types of Evaluation Data That Are Needed 
Another set of issues concerns the types of Chapter 1 participa- 
tion and effectiveness data that are most useful to evaluaf'on users. 
With regard to Chapter 1 participation data, CEP staff report that 
they have regular needs for data on (1) numbers of students receiving 
Chapter 1 instruction in each of the designated instructional and 
supportive areas, (2) participants* grade levels, and (3) to a lesser 
extent, their racial/ethnic affiliation. By contrast, they have vir- 
tually no needs for data on students * age or gender . None of our 
other respondents reported needs for data on students' age or gender. 
Dougherty reports that Chapter 1 coordinators in SEAs find data on 
participants* race, age, and gender to be unnecessary ''since there 
was no allegation of race or sex discrimination in the program; and 
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age data largely duplicated data . . • on the grade level distribu- 
tion of Chapter 1 students*' ( 1986, p. U9). 

With regard to achievement data, congressional staff interviewed 
for this study said that they need information on the educational 
achievement of Chapter 1 participants, and they need it on a regular, 
periodic basis. They said that they do not necessarily need state- 
level data and thus that the results of national sample studies 
administered by ED would be acceptable to them.^ We note, however, 
that members of Congress sometimes ask ED for information on the 
Chapter 1 performance of their Dwn states, suggesting t^^t they might 
be less willing than their staffs to do without state Chapter 1 data. 

A different perspective on what types of effectiveness data are 
needed comes from a senior CEP official. According to this 
individual, the reading and mathematics achievement of Chapter 1 
participants has received excessive attention from policy makers and 
evaluators, to the neglect of other, equally valuable categories of 
outcome information. Types of national effectiveness data that 
should be obtained, according to this respondent, include (1) the 
performance of Chapter 1 participants in instructional areas other 
than those that Chapter 1 addresses, (2) participants' retention in 
school, and (3) their school behavior, including their self-concept 
and disciplinary problems. Other ED staff note, however, that it may 



^ This flexibility contrasts with earlier positions expressed 
by the congressional committees, in which members indicated that they 
^ did not trust the results of federally sponsored national studies 

because of possible political inten'ention to influence the conclu- 
sions reached in the studies (Reisner in Reisner, Alkin, Boruch, 
Linn, & Millman, 1982, p. 89). 
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not be reasonable to expect Chapter 1 to produce measurable outcomes 
in these areas, given the relatively small amounts of supplementary 
instruction that Chapter 1 provides. 
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III. FEDERAL ADMINISTRATION OF CHAPTER 1 PROGRAM IMPROVEMENT 

The major new federal initiative in Chapter 1 since 1981 focuses 
on program improvement. With a relatively small financial invest- 
ment, ED has launched several activities intended to improve the 
classroom experiences of Chapter I's five million participants. In 
this section, we describe (1) the major events since 1981 in the 
federal administration of Chapter 1 program improvement activities, 
(2) the program improvement activities that ED is currently imple- 
menting, and (3) the opportunities and problems characterizing this 
series of initiatives. 

A problem that arises in describing Chapter 1 program improve- 
ment is the difficulty of defining the term precisely. Three 
slightly different meanings that are attached to the term most fre- 
quently are (1) the general upgrading of Chapter 1 instruction, often 
based on principles described in the "effective schools" research; 
(2) the pro\lsion of intensive assistance to Chapter 1 projects hav- 
ing particularly poor records of performance; and (3) the use of 
evaluative information to identify instructional needs and suggest 
strategies for improving the achievement of Chapter 1 students. 

Events Since 1981 in the Federal Administration of Program Improve- 
ment Activities 

To sorae extent, ECIA represented a reaction to the compliance 
model of federal program administration in compensatory education. 
In place of implementing the federal program through regulation and 
penalty, as envisioned in the 1978 amendments to Title I, Chapter 1 
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reflects an emphasis on encouragement and assistance. The program 
improvement initiative has been a centerpiece of that effort, operat- 
ing alongside other federal activities to promote Chapter 1 
compliance . 

In school year ly83-8A, the Department laiinched ''the Secretary's 
Initiative to Improve the Quali'iy of Chapter 1 Projects" with the 
award of 21 one-year grants, totalling $1 million, to support pro- 
jects involving 33 SEAs. ED directed grantees to focus their 
activities on identifying, implementing, and disseminating approaches 
to improve Chapter 1 project quality. To illustrate the types of 
projects funded, a program announcement issued after the <$rant com- 
petition included the following three grant descriptions (selected at 
random) : 



Vireinia ($85,000) 

The application submitted by the Virginia Department of 
Education proposed activities in five states (Va . , Del . , 
Md., Pa., and W. Va.) and the District of Columbia, the 
involvement of two regional laboratories (Appalachia Educa- 
tional Laboratory and Research for Better Schools ) , and 
evaluation support from their Chapter 1 TACs . The consor- 
tium proposed to develop a program improvement guide (the 
Mid-Atlantic Guide for Improving Chapter 1), train 120 
Chapter 1 staff members in its use, field test the guide in 
80 local program sites, and institutionalize the use of the 
guide in the state monitoring process. 



Utah ($12,A92) 

The Utah project is designed to analyze the cost-effec- 
tiveness of parent-child tutoring programs in kindergarten/ 
first-grade reading. The project will: evaluate the cost- 
effectiveness of using parents as tutors; develop proced- 
ures to monitor and document parent training; and produce 
documents for parents and educators. 



Iowa ($69,965) 

The Iowa Department of Public Instruction proposes to iden- 
tify effective instructional program elements in the class- 
room. Over the past three years, TIERS has led to the 
identification of consistently high achieving and consis- 
tently low achieving projects. The amounts and kinds of 
within-class variables will be examined to determine those 
factors of compensatory reading programs which consistently 
contribute to success or failure. 

In the following year ED redirected its program improvement 
activities in Chapter 1 . For school year 198A-85 ED initiated a 
three-part program intended to (1) provide technical assistance on 
Chapter 1 program improvement to all the states through the TACs, (2) 
synthesize and disseminate information regarding successful and cost- 
effective Chapter 1 projects, and (3) identify and publicize par- 
ticularly successful local projects. ED directed the TACs to devote 
25 percent of their federal contract funds to supporting this initia- 
tive. The Department asked SEAs to nominate particularly successful 
local projects from their states, using the TACs to help in evaluat- 
ing projects and developing project descriptions. From this list of 
state nominated projects, a federally appointed panel of outside 
experts selected 116 for special recognition, based on criteria drawn 
from the '^effective schools** research. 

The criteria were developed with vhe participation of Chaptpr 1 
practitioners and experts in compensatory education and cognitive 
research. Because the research focuses on characteristics of 
schools, critics within CEP and the ED Office of Educational Research 
and Improvement have said that it may not be relevant to categorical 
programs , such as Chapter 1 , that serve only some of the students 
within a school and often span activities in several schools. The 



director of CEP reported that CEP is currently monitoring several 
studies that may lead to refinements in the criteria. 

The projects sel >cted for recognition are described in the 
•'Effective Compensatory Education Sourcebook" {Griswold, Cotton, & 
Hansen, 1986), a two-volume compendium of research findings and pro- 
ject descriptions prepared by staff from one of the TACs. Examples 
of Chapter 1 projects selected for recognition include the following, 
summarized from the "Sourcebook" (chosen at random): 



Bloomington (Ind.) Public Schools seventh-grade project 

This project focuses on skill development in reading, 
composition, spelling, spoken language, and geography, 
along with study skills. Instruction includes students' 
use of computers for language arts practice and word 
processing. The project emphasizes (1) the efficient use 
of students' time in class and (2) parental assistance with 
regular instruction and summer learning activities. 



Salisbury (N.C.) Citv Schools reading and language arts 
project 

Pairs of teachers (one Chapter 1 teacher and one regular 
classroom teacher) provide reading and language arts 
instruction in kindergarten through ninth grade using a 
Chapt^?' 1 replacement model . Reading consultants assist 
project teachers in administrative and instructional plann- 
ing. Instruction is provided in small group settings and 
arranged to avoid the personal stigma and other problems 
that sometimes accompany pullout programs. 

Alpine (Utah) School District first- and second-grade 
project 

Specially trained fifth and sixth graders tutor first and 
second graders under the direction of a teacher and counse- 
lor in each participating school. The project focuses on 
developing reading skills and student self-esteem. Parents 
are involved in supporting their children's learning and in 
suggesting curricular improvements. 



Following the announcement of the local projects selected for 
recognition, ED sponsored regional dissemination and training meet- 
ings for SEAs and other activities, SEAs were encouraged to conduct 
similar activities for their LEAs, and, according to PES respondents, 
most did so with assistance from the TACs. 

In school year 1985-86, ED continued the activities initiated in 
the preceding year. The Department again asked SEAs to nominate ef- 
fective Chapter 1 projects for the national recognition program, and 
from this list ED selected 130 projects. This round of nomination 
and selection differed from the preceding year's in three ways — (1) 
ED asked SEAs to consider all compensatory education projects in 
their states, not just Chapter 1 projects, in selecting local nomi- 
nees (2) the Department used a more formal selection process, with 
training for panelists and rating forms; and (3) CEP examined the 
compliance status of nominated projects (especially in light of 
Aguilar v. Felton ) before the panel made its selections . These 
projects will be described in a third volume of the **Sourcebook,** 
which is currently being prepared by the developers of the first two 
volumes . 

ED^s Current Administrative Activities in Program Improvement 

Program improvement activities in Chapter 1 are handled at the 
federal level by personnel in CEP and PES. Within CEP, one person 
works full time on program improvement, with support from a branch 



SEAs nominated only Chapter 1 projects for recognition, 
despite ED's suggestion that they look beyond the federal program. 



chief. PES personnel monitored the state grants ^ assist in the 
selection of the recognition projects, and coordinate training and 
dissemination activities. The PES staff member responsible for the 
TAG contracts monitors the TaCs' work in promoting program improve- 
ment, described in Section IV. 

ED plans to conduct another round of project recognition 
activities in the 1986-87 school year; no decision has been made on 
whether any subsequent round of project recognition activities will 
be held. CEP staff said that they would like SEAs to continue the 
program improvement initiative by developing statewide program 
improvement plans for Chapter 1, and they have directed the TACs to 
work with SEAs to develop these plans. (Indeed, ED asked the TACs 
to include the development of state program improvement plans in 
their most recent letters of agreement with each SEA.) As described 
' / a senior CEP official, the purpose of the plans is to help SEAs 
(1) implement strategies for upgrading the performance of LEAs v;hose 
Chapter 1 projects show particularly low gains and (2) work with all 
Chapter 1 LEAs in designing and implementing programs that reflect 
the findings of research on effective services to educationally dis- 
advantaged children. Because no statutory authority exists requiring 
state Chapter 1 improvement plans, CEP understands that this idea can 
be promoted only on a voluntary basis Even so, CEP staff report 

PES staff concurred with the priority on state improvement 
plans and said they would also like the program to sponsor validation 
reviews of the Chapter 1 projects previously selected for recogni- 
tion. The purpose of these reviews would be to determine (1) whether 
the projects were continuing to be implemented as they were at the 
time they were selected and (2) the effects of each project on par- 
ticipants and on the LEA. 



that 18 states have already developed and adopted such plans (involv- 
ing varying levels of specificity) on their own or with TAG aid. 

CEP^s interest in state improvement plans reflects a desire to 
help SEAs and LEAs move towards increased commitments to program 
improvement goals and methods. As an indicator of movement in this 
direction, CEP staff point to the fact that several state Chapter 1 
policy/procedures manuals now include sections on program improve- 
ment. Some of the states* LEA applications also now include require- 
ments to address the ''effective schools" criteria adopted for the 
Chapter 1 project recognition program. CEP staff were not able to 
comment on the extent to which these activities have led to changes 
in LEA practice . 

To promote improvement goals in CEP's own administrative 
activities, the CEP staff person responsible for progrjim improvement 
has developed a one-page monitoring guide for use by federal Chapter 
1 program review teams. The guide identifies the following areas for 
inquiry during Chapter 1 program reviews of SEAs: 

• Chapter 1 staff^s familiarity with the program 
improvement initiative , 

# SEA^s plan for program improvement, 

• SEA*s p 'ocedures for participating in the Chapter 
1 LEA recogiiition program, 

# SEA' s use of evaluation data to identify low 
achieving projects and potential Joint Dissemina- 
tion Review Panel submis**ions, 

# SEA*s dissemination plan, and 

• SEA*s workshop and conference agenda. 
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Senior CEP staff expect review teams to use the guide when they 
conduct their SEA and LEA visits, and program managers are currently 
revising it on the basis of CEP experience with it to date. We were 
told that the teams decide whether to use the current guide depending 
on (1) the particular interests of the review team members and (2) 
the other priorities to be addressed in the state. 

Opportunities and Problems in the Federal Administration of Program 
Improvement 

The Chapter 1 program improvement initiative has roots in both 
federal legislative action and grass-roots interests in educational 
reform. At the legislative level, ECIA reflected a broad trend 
towards deregulation, burden reduction, and less federal involvement 
in local program activities; the Chapter 1 program improvement 
activities are consistent with that outlook. The Chapter 1 program 
improvement effort also responds to the national trend supporting 
educational reform, a movement that has sparked growing interest in 
schools and classrooms. By adopting some of the goals of this move- 
ment, the Chapter 1 improvement initiative has endeavored to direct 
attention to the quality of instruction provided to Chapter 1 recipi- 
ents, who include many of the nation's neediest students. 

A major problem impeding adoption of Chapter 1 program 
improvement priorities is that instructional improvement has never 
been an explicit component of the SEA*s mission in the administration 
of Chapter 1. Because the SEA role in Chapter 1 has historically 
focused on compliance, most SEAs have staffed their Chapter 1 offices 
mainly with persons skilled in that area. State Chapter 1 offices 
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have been se'-iously limited in their ability to add staff members 
knowledgeable in the program improvement areas highlighted by the 
federal initiative, due to (1) continuing SEA responsibilities for 
Chapter 1 compliance and (2) constraints on Chapter 1 administrative 
spending . 

The limited administrative funds have also impeded SEAs in tak- 
ing advantage of ED's own initiatives. As one state Chapter 1 
coordinator said to us, the Chapter 1 initiatives are "fine to a 
point," but they stop short. "We know where our good piojects are," 
this coordinator said, but "it doesn't do anyone any good to know 
where the other good ones are (in other states), and then give us no 
money to learn any more about them." This respondent said that LEAs 
generally cannot foot the bill for staff and travel costs to visit an 
ED-recognized project. 

A third problem arises from the relationship between some SEAs 
and LEAs. In some instances ^ especially when the LEA has a very 
large enrollment, the LEA may not be willing to accept suggestions or 
assistance from the SEA (according to interviews with state Chapter 1 
coordinators and TAC directors). This is particularly likely to be 
true if the LEA's involvement with the SEA is voluntary, as in the 
case of Chapter 1 program improvement. 

Despite the problems, this relationship between a iederal pro- 
gram initiative and a larger national priority creates special 
opportunities in Chapter 1. In particular, it may make it possible 
to foster strengthened state and local com^^' ""^nts to improving com- 
pensatory education programs. Development of these strengthened 
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commitments and tapping the broader reform objectives of SEAs and 
LEAs represent a major challenge confronting Chapter I's federal 
administrators • 



A report being prepared by B . J. Turnbull discusses the 
relationship etween Chapter 1 and state educational improvement 
initiatives . 
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IV. FEDERAL ADMINISTRATION OF CHAPTER 1 TECHNICAL ASSISTANCE 

ED provides several types of technical assistance to SEAs and 
LEAs implementing Chapter 1 programs. These include technical 
assistance in interpreting and complying with programmatic require- 
ments, such as those concerned with school and student selection and 
the prevision of services to private school students* ED also pro- 
vides technical assistance through telephone consultations, national 
and regional conferences, attendance at state sponsored meetings, and 
sponsorship of the TACs. In this report we focus mainly on the TACs, 
the main federal vehicle for delivering technical assistance in the 
evaluation and qualitative improvement of Chapter 1 programs. 

Initiated in 1976, the TACs have seen their mission change with 
the shifting federal administrative role in evaluation and program 
improvement. The services they provide have changed as well, 
reflecting their evolving mission. Because they serve as a link 
between federal policy in Chapter 1 and state and local practice in 
evaluation and program improvement, their experiences are a useful 
barometer of the extent to which SEAs and LEAs are reacting to and 
acting on federal priorities in Chapter 1. 

In this section, we describe (i) the major changes since 1981 in 
the federal administration of the Chapter 1 TACs, (2) the technical 
assistance ED currently provides, and (3) the opportunities and prob- 
lems characterizing this area of federal Chapter 1 administration. 
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Changes Since 1981 in the Federal Administration of Chapter 1 Techni- 
cal Assistance 

In 1974 Congress charged the Commissioner of Education with pro- 
viding technical assistance to SEAs "to enable them to assist LEAs in 
the development and application of a systematic evaluation of pro- 
grams in accordance with the [required Title I evaluation] models'* 
(Section 151 of the 1978 ESEA amendments)* To implement this 
requirement, the U,S. Office of Education established 10 TACs in 
1976, with each serving a designated geographic region. Dvring the 
decade since, the agenda of the TACs has reflected the events and 
issues shaping Title I and Chapter 1 evaluation. 

With ECIA's repeal of certain key evalua^-ion requirements and 
elimination of the specific authorization for technical assistance 
for evaluation, the role of the TACs shifted somewhat. PES and CEP 
directed the TACs to reduce their emphasis on training SEA and LEA 
staff in che implementation of the evaluation models and to enlarge 
their efforts in assisting SEAs and LEAs to use evaluation results to 
examine and improve their Chapter 1 programs. Changes within CEP 
encouraged this transition away from the TACs* more narrow focus on 
evaluation practices. In the 1970s, CEP had employed program 
specialists to provide technical assistance and other services , such 
as training and materials development , in areas such as basic skills 



^•^ Stonehill and Groves (1983) describe the relationship 
between the TACs' agenda and the events and issues affecting Chapter 
1 evaluation, with a particular emphasis on the role of technical 
assistance in improving the local utility of Chapter 1 evaluations. 
What they characterized as an issue of local utility (i.e., how LEAs 
CBA use evaluations to improve Chapter 1 projects) in 1983 is today a 
component of the Department's focus on program improvement. 
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instruction, educational needs assessment, and parent involvement 
(Funldiouser, Mxchie, & Moore, 1987). Reductions in ED staffing in 
the early 1980s eliminated these positions, though not CEP's respon- 
sibilities in these areas, forcing CEP — and hence SEA Chapter 1 staff 
and evaluators — to turn to the TACs for technical assistance in areas 
other than evaluation. 

Other changes in the TACs resulted in part from the studies of 
TAC operations conducted for ED. The first study (Millman, Paisley, 
Rogers, Sanders, & Womer, 1978) found that the technical assistance 
program was "working and working well" (p. 57). The report recom- 
mended certain changes concerned mainly with improving the federal 
administration of the TACs. The recommendations included suggestions 
to (1) extend the TAC funding cycle from one to three years, (2) 
expand the s.:ope cf lAC activity to in. lude technical assistance for 
local program improvement, (3) establish a uniform system of 
cost/effort reportinj, and (A) encourage cooperata'm among T^Cs. In 
the next cycle of TAC awards, stP'^s were taken to impleinent the 
recommended changes. 

A 1979 Fast Response Survey of LEAs conducted by the Na^'onal 
Center for Education Statistics reinforced the recommendation of 
Millman et al. to include a new TAC focus on program improvement 
(Goor, 1979). The survey asked LEAs about their expected needs for 
technical assistance in conducting Title I evaluations. The are.~ in 



In response to the third recommendation, federal project 
officers instituted ^ reporting system whereby TACs report quarterly 
on their client contacts, categorizing their contacts according to 
service method and topic (see Tables 2 and 3). 



which the highest percent of LEAs (34 percent) reported ne^ds was 



"evaluation for continuing program improvement." 

A second major study of the TACs (Millman in Reisner et al., 
1982) reported four major findings: 

• The content of the assistance that TACs provided 
had shifted from implementing^ the RMC models and 
TIERS reporting system tu improving the quality 
and utility of Chapter 1 evaluation data. 

• States varied widely in the amount of TAC ser- 
vices they received, and TACs varied widely in 
the amount and types of services they provided. 

• The amount o^ field service TACs provided was low 
relative to TAC expenditures . 

• TAC clients were very satisfied with TAC services 
and wanted them to continue. 

At about the same time as this report — though unrelated to it — 

ED spending on the TACs dropped. Table 1 display? total ED spending 

for the TACs since their inception and the percent of total Title I 

or Chapter 1 spending represented by the TACs. 



Table J 

ED Spending on the TACs, 1976-77 through 1986-87 



Percent of Title 1/ 

School Year Amount Chapter 1 Spending 



1976-77 


$1,590,845 


0.1% 


1977-78 


4,087,6A4 


0.2 


1978-79 


5,297,119 


0.2 


1979-80 


6,648,639 


0.2 


1980-81 


8,508,0S0 


0.3 


1981-82 


8,260,388 


0.3 


1982-83 


5,286,?32 


0.2 


1983-8^ 


5,596,979 


0.2 


1984-85 


4,170,000 


0.1 


1985-86 


3,6r.5,000 


0.1 


1986-87 


3,600,000 


0.1 



The TAG reductions since 1980-81 reflect both overall cutbacks 
in Chapter 1 allocations and also specific reductions in TAG funding. 
This is demonstrated by the fact that TAG spending has declined as a 
percent of total Ghapter 1 c))propriatio;"is since 1981, at the same 
time as the absolute amount of TAG spending has decreased. According 
to respondents in PES, current TAG staffing overall stands at 29 
full-time equivalent (FTE) staff members. 

The types of activities undertaken by the TAGs reflect the 
reductions in TAG funding. Analysis of self-reported data from the 
TAGs, presented in Table 2, indicates, that the percentage of TAG con- 
tacts with SEA and LEA clients made through workshops and on-site 
consultations has decreased since 1982, and the percentage of con- 
tacts made through telephone calls has increased. The percentage 
made through correspondence has fluctuated, increasing in the most 
recent year. This pattern reflects the TAGs' adoption of less costly 
means of conveying assistance, as a replacement for personnel- and 
time-intensive workshops and on-site consultations. 

Table 2 

Percent of TAG Contacts by Service Delivery Method, 
1982-83 through 1985-86 



Method 1982-83 1983-84 1984-85 1985-86 ^ 



Workshops 


13% 


11% 


10% 


6% 


On-site consultations 


22 


2A 


20 


11 


Telephone calls 


37 


Al 


47 


52 


Correspondence 


28 


2A 


23 


31 



Data for 1985-86 are incomplete. Percents are based on data 
from October 1985 through June 1986. 
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The types of issues addressed through TAG services have &lso 
changed during the same period, as described -m Table 3, As might be 
expected due to shifts in federal requirements and growing local 
familiarity with Chapter 1 evaluation activities, services concerned 
with evaluation procedures, uses, and reporting declined as a per- 
centage of all contacts with clients, whilo the percentage of sei - 
vices concerned with program improvement and testing issues 
increased. The percent of TAG contacts related to technology use 
(e.g., uses of mi croc Tiputers for Chapter 1 administrative and 
instructional purposes) also decreased, while the percent of contacts 
on data quality rose and then declined. 

Table 3 

Percent of TAG Contacts by Service Topic, 
1982-83 through 1985-86 



Service Topic 


1982-83 


1983-8A 


198A-85 


1985-86^ 


Evaluation 










procedures 


37% 


26% 


27% 


29% 


Evaluation uses 










and reporting 


14 


10 


8 


9 


Program 










improvement 


10 


18 


25 


30 


Testing issues 


19 


21 


21 


23 


Technology uses 


14 


U 


7 


2 


Data quality 


6 


11 


12 


7 


3 Data for 1985-86 
data from October 


are incomplete. Percentages 
1985 through June 1986. 


are based on 
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ED^s Current Administrative Activities in Chapter 1 Technical 
Assistance 

PES respondents report that they devote 0.5 FTE to administering 
the TAC contracts. Their ad;ninistrative responsibilities r^^gularly 
include (1) reviewing TAC plans and products, (2) answering questions 
from the TACs , ( 3 ) responding to SEA inquiries regarding the TACs , 
and (A) attending TAC meetings. 

All ED and TAC staff we questiOiied regarding current TAC activi- 
ties reported that ED continues to require the TACs to focus services 
on improving the quality of Chapter 1 instructional programs. The 
TAC directors said that at least one-half of their total resources 
are used for services in this area. A recent list of new workshops 
and services publicized by one TAC indicates this emphasis, as 
follows : 

Research on reading comprehension. 
Effective Chapter 1 leadership. 
Early childhood tests. 

Coordination of the regular and Chapter 1 program. 

Improving study sLills (workshop for teachers and 
parents), and 

Program improvement in Chapter 1. 
Only one of the topics on the list — early childhood tests — is 
concerned solely with evaluation. 

In their implementation of the program improvement agenda, the 
TACs are serving as a communications channel between ED and state and 
local educational agencies, according to CEP and SEA respondents. 
Indeed, much of ED's focus on program improvement has been 



3 



39 



implemented through the TACs, rather than through regulations, non- 
regulatory guidance, or direc\:ives. For example, ED is encouraging 
TACs to work with SEAs to identify LEAs with particularly poor 
records of Chapter 1 performance, in order to assist these LEAs to 
assess and upgrade their program activities. Senior CEP and PES 
officials and TAG directors understand that these intensive services 
to a small number of LEAs appear relatively inef/.icient in TAG 
measures of clirnt contacts. Nevertheless, they believe that these 
services are central to improving Ghapter 1 performance. 

Our interviews with TAG directors indicated that working with 
SEAs to identify and assist poorly performing LEAs has proved a dif- 
ficult assignment. The only SEAs prepared to direct TACs tow;^rds 
LEAs with poor performance records are those fev' SEAs thac have 
already adopted statewide improvement programs and are targeting 
poorly performing LEAs as part of that effort. Other SEAs are reluc- 
tant to iaentify any LEAs as having ineffective Ghapter 1 prograi.iS . 
One TAG director described an especially difficult situation in which 
the SEA asked ^'he TAG to work with several LEAs with poor performance 
records but would not tell the LEAs why they had been selected to 
receive special TAG help. In this instance and in other similar 
situations in which the SEA has selected an LEA to receive special 
help, TAC directors report difficulties in securing cooperation from 
the LEA and sometimes a lack of interest in the services the TAG can 
provide. 

As the example indicates, the TACs have found themselves romot- 
ing the federal program improvement initiative, which has minimal 
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statutory authority, in SEAs with varying interests and authority. 
If an SEA is reluctant to adopt Chapter 1 program improvement priori- 
ties, the TAG can do little beyond persuasion and coaxing. Similar- 
ly, if SEA staff are committed to program improvement bet hesitate to 
direct their districts to improve their Chapter 1 programs, the TAC's 
effectiveness will be limited. 

Several ED staff members said in interviews that some TACs are 
not well equipped to offer pr(,gram improvement services. They 
expressed a perception that some TAG sta^f 'nembers are iinCv'mf ortable 
providing t3chnical assistance on program improvement., mainly because 
their training and expertise are primarily in evaluation . PES 
staff responsible for supervising the TAG contracts said that TAG 
resumes were reviewed as part of the selection of TAG contractors to 
ensure that staff members in each TAG are qualified to assist in 
program improvement as well as in evaluation and testing. PES staff 
also report that they review the resumes of new staff to determine 
whether they have training or work experience in ar^ uch as read- 
ing Instructional resign, curriculum, ano administration. 

In our interviews, TAG directors said that their staff are com- 
petent to offer technical assistance in program improvement, primari- 
ly because of their work experiences, interests, and familiarity with 
state and local staff . Indicating the difficulty of ED's job in 



This assessment is supported by the fact that, for the most 
part, the same organizations have operated the TAGs since the reduc- 
tion from ten to four TAGs in 1982, even though the organizational 
umbrellas for several TAGs have shifted since 1982. Moreover, the 
four TAGs chosen in 1982 were essentially consortia of organizations 
(and individuals) operating TAGs under the previous configuration of 
ten TAGs, 
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ass:jssing the qualifications of TAG staff, one TAG director said, 
know our people are trained as evaluators and have done lots of eval- 
uations. I a.lso know the people on our staff who do really good 
program improvement. But there's no such thing as a master's in 
program improvement, so I'm not sure what [ED staff) are going to be 
1 ^icing for on resumes." 

Based on his interviews with 50 state Ghapter 1 coordinators, 
Dougherty ( 1 985 ) concludes that "SEAs consider the TAGs to be an 
extremely valuable resource in improving Ghapter 1 quality" (p. 165). 
Dougherty raises a concern, however, that ED's program improvement 
initiative is competing with the SEAs' own priorities for scarce TAG 
resources; prompted by this concern, he urges an increase in overall 
TAG funding. 

Differing levels of state interest vnd authority and of TA^^ 
staff abilities have produced variations in TAG program improvement 
activities, both across the TAGs and within the region a single 'AG 
covers. The following types of program improvement activities, drawn 
from our interviews with TAG staff , provide examples of the 
variation: 

• In one state, where the SEA exercises little authority 
over local operations, the TAG conducts workshops to 
introduce evaluation purposes , methods , and require- 
ments to LEA staff who have not previously implemented 
evaluations. As part of the workshops, TAG staff 
discuss how local staff can use their evaluation 
results to improve their Chapter 1 projects. 

• Another TAG helped a state co identify four strong and 
four weak districts. TAG staff developed instruments 
to survey three broad areas — coordination between 
Ghapter 1 and the regul? program, curriculum, and 
parental involvement. The survey results sr^ being 
used to assess characteristics of the weak and strong 
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districts. The TAG anticipates using the analysis to 
guide their work with weak districts in imp:, oving 
theix Chapter 1 projects. 

• Another TAG disseminates extensive information about 
program improvement (e.g., how to improve reading 
instruction, how to increase students ' time on task, 
and how and when to use observational instruments) and 
nas offered program improvement workshops on about 50 
different topics. Only one state in this region has 
become actively involved in program improvement, how- 
ever. Although this SEA has a reputation for 
emphasizing Ghapter 1 compliance rather than program 
content, it has identified six low performing 
districts and has asked the TAG to provide special 
help to those districts. The SEA has also developed a 
comprehensive data base cn its Ghapter 1 projects, and 
with data { and some new staff ) the agency's interest 
and activities in program improvement have grown 
quickly. 

• A fourth TAG has hosted a number of workshops through- 
out its region, attends state sponsored meetings to 
make presentations on program improvement , and shares 
information across states and districts. This TAG has 
been less successful in obtaining SEA cooperation to 
focus on low performing districts. The director said, 
**When [ED] firft told us to do it, we tried to obey. 
We're good soldiers. We went out and approached the 
states . We hit a lot of resistance . Then we 
realized, *why waste resources on something tl^at will 
have zero impact?' No one will improve unless they 
really want to.*' Only one state in this region has 
tried to target improvement efforts at low performing 
districts. In this state, SEA staff searched a three- 
yc*ar file to identify the districts that were achiev- 
ing significantly below the state's avo/age; the SEA 
then asked the TAG to provide assis trance to those 
districts. The TAG director said, "Each district was 
coerced into participating. One remains absolutely 
nonenthusiastic 

Several oi the Ghapter 1 coordinators and TAG directors we 
interviev/ed sai^'^ that program improvement will become an integral 
part of Ghapter 1 operations only if the federal government takes a 
more direct role. Their recommendations include (1) adding statutory 
language requiring SEAs to conduct program improvement activities. 
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(2) designating certain Chapter 1 funds specif ic^^lly for program 
improvement,^^ and (3) maintaining the distinction for TAG activities 
that separates evaluation and program improvement. 

The program -Improvement emphasis has not eliminated the TACs* 
work in Chapter 1 evaluation, however. Indeed, ECIA's reduction in 
the SEA admini trative set-aside led some SEAs to reduce their in- 
house capacity in Chapter 1 evaluation, hence prompting greater 
reliance on TAC services. (For some SEAs, evaluation v;as an easy 
area to cut, because they knew they could turn to the TACs to fill 
in.) TAC directors report that, in comparison with previous years, 
SEAs are more likely now to request in-depth assistance in all phases 
of Chapter 1 evaluation and reporting, services that TACs may provide 
under their contracts with ED. In addition, SEAs are more likely to 
rely on the TACs to carry out all of the training and supervision of 
local Chapter 1 evaluation personnel, according to the TAC directors. 

Opportunities and Problems in the Federal Administration of Chapter 1 
Technical Assistanc e 

One view of the TACs * ef f e^. tiveness is available from a 198A 
.survey of state Chapter 1 coordinators conducted by the coordinators' 
national association (National Association of State Chapter 1 Coor- 
dinators, n.d.). The A8 coordinators responding to the survey 
reported high levels of satisfaction with the services the TACs pro- 
vide. In general, they reported greater satisfaction ^ixh the TACs* 



In discussions with ED staff, some Chapter 1 state and loci- 
coordinators have recommended statutory changes that would earmark 
Chapter 1 reallocation funds for program improvement. 
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assistance in improving evaluation quality (58 percent riited them 
"very effective" and 35 percent said they were "okay") than with 
their help in program improvement (33 percent said they were "very 
effective" and A6 percent said they were "okay"). Overall, the state 
coordinators found the TACs to be responsive in their delivery of 
services and sensitive to state needs. 

Survey respondents identified four areas they expected would 
generate the greatest upcoming needs for assistance. These were pro- 
gram improvement evaluation, quality control, information on appro- 
priate testing procedures, and assistance in completing Chapter 1 
evaluations . 

While reflecting a positive view of the TACs, the survey results 
point out the continuing diversity in SEA needs for technical assis- 
tance. Given the limited resources available for the provision of 
TAG services and reports of SEAs' growing reliance on the TAGs to 
cany out evaluation training and data reporting, ED may need to 
establish more stringent priorities for the expenditure of TAG 
resources . 

In addition, clarif t cacion may be needed in defining "program 
improvement" under Ghapter 1 and the appropriate limits to activities 
in this area. For example, are there points at which assistance with 
Ghapter 1 program improvemi"^n^^ runs afoul of the Department of Educa- 
tion Organization Act (Section 103), which prohibits the federal 
"exercise [of j any direction . . . over Lhe curriculum, program of 
instruction, administration, or personnel of any educational institu- 
tion. . ." ? Goncern over this provision (and a similar provision in 



the General Education Provisions Act prior to the formation of ED) 
deterred TAG involvement in program improvement activities before ED 
launched its Chapter 1 improvement initiative in 1983.^^ 

At the same time the categorical nature of Chapter 1 has limited 
the TACs in the types of assistance they can provide in program 
improvement. Although much recent research on improving instruction- 
al programs focuses on school characteristics and activities, Chapter* 
1 projects only rarely involve the whole school. Because the 
projects usually involve only a portion of a school's students and 
stafi and because TACs are authorized only to assist in improving 
Chcpter 1 programs, TACs are constrained in the scope and complete- 
ness of the assistance they can provide. A solution recommended by 
one person interviewed for this study would be to authorize TACs to 
provide techn:.:al assistance on improving services to Chapter 1 stu- 
dents , assistance that might extend beyond the Chapter 1 program 
itself. Under this proposal, TACs could provide technical a-'^sistance 
in improving any educational services provided to Chapter 1 partici- 
pants, whether the services were supported by Chapter 1 or not. 



CEP staff said that TAC assistance entails no -direction" 
over educational activities at all but only assistance and sometimes 
pressure to implement Chapter 1 projects in accordance with the law. 
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V, CONCLUSIONS 

The preceding description of the federal administration of 
Chapter 1 evaluation, program improvement, and technical assistance 
indicates an ambitious array of federally prompted activities imple- 
mented in a changing environment. These factors combine to suf^gest 
that, however desirable the goals of these federal administrative 
activities may be, their successful implementation will not be easy. 
Indeed, any assessment of their success, which must measure the ef- 
fects of these federal activities on local program implementation, 
should take account of the time and other resources needed to 
accomplish the voluntary changes they envision. 

Our examination of the federal implementation of these adminis- 
trative changes indicates certain areas where it is particularly 
important to learn if the voluntary strategies are working. One such 
area is the state and local collection and reporting of Chapter 1 
achievement data. Now that SEAs and LEAs are no longer required to 
:'.ollow regulated procedures for collecting and reporting these data, 
it would be useful to assess the quality of ^e data that the states 
are providing to ED and the extent to which SEAs and LEAs are making 
use of these data on their own. If the data are of poor quality or 
if SEAs and LEAs make little use of them, ED should consider dropping 
their request that TIERS data be submitted in fulfillment of states* 
evaluation responsibilities. The system is too costly to be con- 
tinued out of habit alone. 



56 



kl 



As part of the review of states' reporting requirements, it 
would be useful to determine if all the current reporting categories 
on Chapter 1 participants are indeed necessary. Our interviews 
suggested that three currtnc categories produce ubefu] data — 
participates ' grade level, Chapter 1 services, and racial/ethnic 
group. Data on participants' age and gender appeared less useful at 
either federal or state levels. 

A second area where it wou^d be useful to determine if current 
approaches are working is in the federal attempt to link Chapter 1 
improvement activities to .he larger educational reform movement. If 
states and local school districts are indeed incorporating compen- 
satory education goals in'.o their plans and activities for improving 
instructional services generally, then it is reasonable to expect 
that "he initiative and ideas for improvement in compensatory educa- 
tion can come from SEAs and LEAs and that the federal role can con-- 
tinue to emphasize financial support and encouragement, rather an 
requirement. If, however, disadvantaged students are receiving less 
than their full share of attention under educational reform — or if 
they are being slighted by some SEAs and LEAs though not others — the 
current voluntary strategies may not be adequate, and some stronger 
requirements may need to be adopted in Chapter 1. 

The "hird area for furt^^r consideration is possible refinements 
in the TACs' responsibilities. At present, SEAs decide on the mix of 
evaluation and program improvement assistance they receive from the 
lACs and the content of that assistance, with some encouragenent from 
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the TACs and ED. Given current resource constraints, ED may want to 
establish more explicit priorities for TAG assistance. 

We consider the federal administrative activities described in 
this report to be ajnbitious because they are intended to promote the 
voluntary upgrading of narrowly defined services delivered to a needy 
population. These changes from the federal implementation of Title I 
have meant that ED is relying on state and local intentions '.o assess 
and improve federal compensatory education services to a much greater 
degree than before. Though ED has provided some new resources to 
help SEAs and LEAs make these improvements (e.g., a year's worth of 
small SEA grants, special services through the TAGs, certain types of 
regulatory relief), the new resources have not been extensive, 
especially in light of Ghapter I's status as the nation's largest 
categorical prograjn in elementary and secondary education. 
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